{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# =============================================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train tensorflow or keras model in .py files on GCP or Kubeflow from Notebooks\n",
    "\n",
    "This notebook introduces you to using Kubeflow Fairing to train the model, which is developed using tensorflow or keras and enclosed in python files, to Kubeflow on Google Kubernetes Engine (GKE), and Google Cloud AI Platform training. This notebook demonstrate how to:\n",
    " \n",
    "* Use Kubeflow Fairing to train an Tensorflow model remotely on Kubeflow cluster,\n",
    "* Use Kubeflow Fairing to train an Tensorflow model remotely on AI Platform training,\n",
    "\n",
    "**You need Python 3.6 to use Kubeflow Fairing.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setups\n",
    "\n",
    "* Pre-conditions\n",
    "    - Deployed a kubeflow cluster through https://deploy.kubeflow.cloud/\n",
    "    - Have the following environment variable ready: \n",
    "        - PROJECT_ID # project host the kubeflow cluster or for running AI platform training\n",
    "        - DEPLOYMENT_NAME # kubeflow deployment name, the same the cluster name after delpoyed\n",
    "        - GCP_BUCKET # google cloud storage bucket\n",
    "\n",
    "* Create service account\n",
    "```bash\n",
    "export SA_NAME = [service account name]\n",
    "gcloud iam service-accounts create ${SA_NAME}\n",
    "gcloud projects add-iam-policy-binding ${PROJECT_ID} \\\n",
    "    --member serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \\\n",
    "    --role 'roles/editor'\n",
    "gcloud iam service-accounts keys create ~/key.json \\\n",
    "    --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\n",
    "```\n",
    "\n",
    "* Authorize for Source Repository\n",
    "```bash\n",
    "gcloud auth configure-docker\n",
    "```\n",
    "\n",
    "* Update local kubeconfig (for submiting job to kubeflow cluster)\n",
    "```bash\n",
    "export CLUSTER_NAME=${DEPLOYMENT_NAME} # this is the deployment name or the kubenete cluster name\n",
    "export ZONE=us-central1-c\n",
    "gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${ZONE}\n",
    "```\n",
    "\n",
    "* Set the environmental variable: GOOGLE_APPLICATION_CREDENTIALS\n",
    "```bash\n",
    "export GOOGLE_APPLICATION_CREDENTIALS = ....\n",
    "```\n",
    "```python\n",
    "os.environ['GOOGLE_APPLICATION_CREDENTIALS']=...\n",
    "```\n",
    "\n",
    "* Install the lastest version of fairing\n",
    "```python\n",
    "pip install git+https://github.com/kubeflow/fairing@master\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please not that the above configuration is required for notebook service running outside Kubeflow environment. And the examples demonstrated in the notebook is fully tested on notebook service outside Kubeflow cluster also.**\n",
    "\n",
    "**The environemt variables, e.g. service account, projects and etc, should have been pre-configured while setting up the cluster.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import fairing\n",
    "from fairing.cloud import gcp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting up google container repositories (GCR) for storing output containers\n",
    "# You can use any docker container registry istead of GCR\n",
    "# For local notebook, GCP_PROJECT should be set explicitly\n",
    "GCP_PROJECT = fairing.cloud.gcp.guess_project_name()\n",
    "GCP_Bucket = os.environ['GCP_BUCKET'] # e.g., 'gs://kubeflow-demo-g/'\n",
    "\n",
    "# This is for local notebook instead of that in kubeflow cluster\n",
    "# os.environ['GOOGLE_APPLICATION_CREDENTIALS']="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In this demo, I use gsutil, therefore i compile a special image to install GoogleCloudSDK as based image\n",
    "base_image = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)\n",
    "!docker build --build-arg PY_VERSION=3.6.4 . -t {base_image}\n",
    "!docker push {base_image}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gojek-kubeflow\n"
     ]
    }
   ],
   "source": [
    "GCP_PROJECT = fairing.cloud.gcp.guess_project_name()\n",
    "DOCKER_REGISTRY = 'gcr.io/{}/fairing-job-tf'.format(GCP_PROJECT)\n",
    "BASE_IMAGE = 'gcr.io/{}/fairing-predict-example:latest'.format(GCP_PROJECT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = 'model.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the training job to AI platform training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using preprocessor: <fairing.preprocessors.base.BasePreProcessor object at 0x11f4fe128>\n",
      "Using builder: <fairing.builders.docker.docker.DockerBuilder object at 0x11f4fee48>\n",
      "file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py\", line 36, in autodetect\n",
      "    from google.appengine.api import memcache\n",
      "ModuleNotFoundError: No module named 'google.appengine'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py\", line 33, in <module>\n",
      "    from oauth2client.contrib.locked_file import LockedFile\n",
      "ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py\", line 37, in <module>\n",
      "    from oauth2client.locked_file import LockedFile\n",
      "ModuleNotFoundError: No module named 'oauth2client.locked_file'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py\", line 41, in autodetect\n",
      "    from . import file_cache\n",
      "  File \"/Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py\", line 41, in <module>\n",
      "    'file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth')\n",
      "ImportError: file_cache is unavailable when using oauth2client >= 4.0.0 or google-auth\n",
      "URL being requested: GET https://www.googleapis.com/discovery/v1/apis/ml/v1/rest\n",
      "Building image using docker\n",
      "Docker command: ['python', '/app/model.py']\n",
      "Creating docker context: /tmp/fairing_context_6ulb5qmv\n",
      "Context: /tmp/fairing_context_6ulb5qmv, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py\n",
      "Context: /tmp/fairing_context_6ulb5qmv, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py\n",
      "Context: /tmp/fairing_context_6ulb5qmv, Adding model.py at /app/model.py\n",
      "Context: /tmp/fairing_context_6ulb5qmv, Adding requirements.txt at /app/requirements.txt\n",
      "Context: /tmp/fairing_context_6ulb5qmv, Adding /tmp/fairing_dockerfile_ju8v9_bp at Dockerfile\n",
      "Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E...\n",
      "Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest\n",
      "Build output: \n",
      "Build output: ---> 07b0c0a773a2\n",
      "Build output: Step 2/7 : WORKDIR /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e38aad2dc182\n",
      "Build output: Step 3/7 : ENV FAIRING_RUNTIME 1\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 597bd070338a\n",
      "Build output: Step 4/7 : COPY /app//requirements.txt /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 05e78d5eb908\n",
      "Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e31aa3ffcc59\n",
      "Build output: Step 6/7 : COPY /app/ /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 710caec21dce\n",
      "Build output: Step 7/7 : CMD python /app/model.py\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> d52847d8c0d9\n",
      "Push finished: {'ID': 'sha256:d52847d8c0d9ba6599a6f0f8c85a21fbe0d03ac17093ec6792ed92ab82dfe5fb'}\n",
      "Build output: Successfully built d52847d8c0d9\n",
      "Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E\n",
      "Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E...\n",
      "Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: F0D4918E: digest: sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157 size: 3472 None\n",
      "Push finished: {'Tag': 'F0D4918E', 'Digest': 'sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157', 'Size': 3472}\n",
      "URL being requested: POST https://ml.googleapis.com/v1/projects/gojek-kubeflow/jobs?alt=json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating training job with the following options: {'jobId': 'fairing_job_5310fcf1', 'trainingInput': {'masterConfig': {'imageUri': 'gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:F0D4918E'}, 'region': 'us-central1'}}\n",
      "Job submitted successfully.\n",
      "Access job logs at the following URL:\n",
      "https://console.cloud.google.com/mlengine/jobs/fairing_job_5310fcf1?project=gojek-kubeflow\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(<fairing.preprocessors.base.BasePreProcessor at 0x11f4fe128>,\n",
       " <fairing.builders.docker.docker.DockerBuilder at 0x11f4fee48>,\n",
       " <fairing.deployers.gcp.gcp.GCPJob at 0x11f4fefd0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fairing.config.set_preprocessor('python', executable=file_name, input_files=[file_name, 'requirements.txt'])\n",
    "fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY, base_image=BASE_IMAGE, push=True)\n",
    "fairing.config.set_deployer(name='gcp')\n",
    "fairing.config.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the training job to kubeflow cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using preprocessor: <fairing.preprocessors.base.BasePreProcessor object at 0x11f5e9cf8>\n",
      "Using builder: <fairing.builders.docker.docker.DockerBuilder object at 0x11f5e95c0>\n",
      "Building image using docker\n",
      "Docker command: ['python', '/app/model.py']\n",
      "Creating docker context: /tmp/fairing_context_f3f4quru\n",
      "Context: /tmp/fairing_context_f3f4quru, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/__init__.py at /app/fairing/__init__.py\n",
      "Context: /tmp/fairing_context_f3f4quru, Adding /Users/luoshixin/LocalSim/virtualPython36/lib/python3.6/site-packages/fairing/runtime_config.py at /app/fairing/runtime_config.py\n",
      "Context: /tmp/fairing_context_f3f4quru, Adding model.py at /app/model.py\n",
      "Context: /tmp/fairing_context_f3f4quru, Adding requirements.txt at /app/requirements.txt\n",
      "Context: /tmp/fairing_context_f3f4quru, Adding /tmp/fairing_dockerfile_wpuoc5id at Dockerfile\n",
      "Building docker image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:BA147BAB...\n",
      "Build output: Step 1/7 : FROM gcr.io/gojek-kubeflow/fairing-predict-example:latest\n",
      "Build output: \n",
      "Build output: ---> 07b0c0a773a2\n",
      "Build output: Step 2/7 : WORKDIR /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e38aad2dc182\n",
      "Build output: Step 3/7 : ENV FAIRING_RUNTIME 1\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 597bd070338a\n",
      "Build output: Step 4/7 : COPY /app//requirements.txt /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 05e78d5eb908\n",
      "Build output: Step 5/7 : RUN if [ -e requirements.txt ];then pip install --no-cache -r requirements.txt; fi\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> e31aa3ffcc59\n",
      "Build output: Step 6/7 : COPY /app/ /app/\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> 710caec21dce\n",
      "Build output: Step 7/7 : CMD python /app/model.py\n",
      "Build output: \n",
      "Build output: ---> Using cache\n",
      "Build output: ---> d52847d8c0d9\n",
      "Push finished: {'ID': 'sha256:d52847d8c0d9ba6599a6f0f8c85a21fbe0d03ac17093ec6792ed92ab82dfe5fb'}\n",
      "Build output: Successfully built d52847d8c0d9\n",
      "Build output: Successfully tagged gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:BA147BAB\n",
      "Publishing image gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job:BA147BAB...\n",
      "Push output: The push refers to repository [gcr.io/gojek-kubeflow/fairing-job-tf/fairing-job] None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Preparing None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Waiting None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: Layer already exists None\n",
      "Push output: BA147BAB: digest: sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157 size: 3472 None\n",
      "Push finished: {'Tag': 'BA147BAB', 'Digest': 'sha256:55679cf333fc88efc221d4ba7cede8d88658d210c7ec045bda3160199575a157', 'Size': 3472}\n",
      "Training job fairing-job-s6pfs launched.\n",
      "Waiting for fairing-job-s6pfs-wr9gb to start...\n",
      "Pod started running True\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:TF_CONFIG {}\n",
      "INFO:tensorflow:cluster=None job_name=None task_index=None\n",
      "INFO:tensorflow:Will export model\n",
      "WARNING:tensorflow:From /app/model.py:179: load_mnist (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:300: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use urllib or similar directly.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tensorflow/logs', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fdeb8464390>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Not using Distribute Coordinator.\n",
      "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
      "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_queue_runner.py:62: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "To construct input pipelines, use the `tf.data` module.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/inputs/queues/feeding_functions.py:500: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "To construct input pipelines, use the `tf.data` module.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /app/model.py:87: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.conv2d instead.\n",
      "WARNING:tensorflow:From /app/model.py:89: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.max_pooling2d instead.\n",
      "WARNING:tensorflow:From /app/model.py:105: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n",
      "WARNING:tensorflow:From /app/model.py:109: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dropout instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "2019-05-10 05:58:07.339398: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2019-05-10 05:58:07.346743: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
      "2019-05-10 05:58:07.348307: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x43f44b0 executing computations on platform Host. Devices:\n",
      "2019-05-10 05:58:07.348362: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py:809: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "To construct input pipelines, use the `tf.data` module.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tensorflow/logs/model.ckpt.\n",
      "INFO:tensorflow:loss = 2.316252, step = 1\n",
      "INFO:tensorflow:global_step/sec: 6.55602\n",
      "INFO:tensorflow:loss = 2.0959373, step = 101 (15.253 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 200 into /tmp/tensorflow/logs/model.ckpt.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-10T05:58:38Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tensorflow/logs/model.ckpt-200\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-10-05:58:38\n",
      "INFO:tensorflow:Saving dict for global step 200: accuracy = 0.8046875, global_step = 200, loss = 0.94937694\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 200: /tmp/tensorflow/logs/model.ckpt-200\n",
      "INFO:tensorflow:Performing the final export in the end of training.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:205: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['classes', 'serving_default']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tensorflow/logs/model.ckpt-200\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: /tmp/tensorflow/model/temp-b'1557467918'/saved_model.pb\n",
      "INFO:tensorflow:Loss for final step: 0.95952034.\n",
      "WARNING:tensorflow:Directory b'/tmp/tensorflow/model/1557467918' already exists; retrying (attempt 1/10)\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Predict: ['classes', 'serving_default']\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
      "INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
      "INFO:tensorflow:Restoring parameters from /tmp/tensorflow/logs/model.ckpt-200\n",
      "INFO:tensorflow:Assets added to graph.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: /tmp/tensorflow/model/temp-b'1557467919'/saved_model.pb\n",
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting /tmp/tensorflow/input_data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting /tmp/tensorflow/input_data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting /tmp/tensorflow/input_data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting /tmp/tensorflow/input_data/t10k-labels-idx1-ubyte.gz\n",
      "Train and evaluate\n",
      "Training done\n",
      "Export saved model\n",
      "Done exporting the model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cleaning up job fairing-job-s6pfs...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(<fairing.preprocessors.base.BasePreProcessor at 0x11f5e9cf8>,\n",
       " <fairing.builders.docker.docker.DockerBuilder at 0x11f5e95c0>,\n",
       " <fairing.deployers.job.job.Job at 0x120600630>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fairing.config.set_preprocessor('python', executable=file_name, input_files=[file_name, 'requirements.txt'])\n",
    "fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY, base_image=BASE_IMAGE, push=True)\n",
    "fairing.config.set_deployer(name='job')\n",
    "fairing.config.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "virtualPython36",
   "language": "python",
   "name": "virtualpython36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
